Although it may potentially be used in many other applications, the present invention was created in response to a need in the computer/video games industry. In the past several years there has been a dramatic growth in the computer/video games industry which has been due primarily to the increased availability and technical advances of personal computers and video entertainment technology. The price/performance of personal computers and video entertainment technology has improved to a point where they can be found in virtually every home and business. Additionally, the quality and realism of computer game images is rapidly approaching that of film or camera-generated pictures, further enhancing the enjoyment and excitement of the games.
In fact, in certain elementary respects a modern computer game is similar to a motion picture. Both a motion picture and a computer/video game consist of a sequence of images which are viewed using some form of imaging medium. Both use a sound track comprised of music and sound effects to accentuate the feel and mood of the action and storyline. However, the story or action in many of the more sophisticated computer/video games can be changed interactively by a user so that the outcome of the game is determined partially or completely by the user. Thus, the computer game is essentially comprised of a number of interwoven plot or action segments, with the overall experience assembled from these segments.
For example, the computer game "Wing Commander", available from Origin Systems, Austin, Texas, is an interactive space adventure in which the user plays the part of one of the characters of the story. Accordingly, the user engages in dialogue and action with other game characters by making appropriate choices from menus of options, and by directing the flight of a spaceship in space combat. Whether or not the universe is saved from evil forces is determined by the menu choices selected by the user and the flight skill of the user. However, the aesthetic coordination of the music and sound effects with the game action in Wing Commander, as in other existing games, results in many instances of random musical juxtapositions which often sound abrupt and unnatural rather than musically graceful. Wing Commander is representative of the current state of the art of computer game technology.
Although music and sound effects are an important part of the game's "feel," the technological progress which has been made in this area has been relatively limited. The use of technology from the music industry, such as synthesizers and the Musical Instrument Digital Interface (MIDI), has yielded an increase in the quality of the composition of music in computer entertainment systems, however, there has been little technological advancement in the intelligent control needed to provide automated music composition which changes gracefully and naturally in response to dynamic and unpredictable actions or the "plot" of the game.
The most advanced music systems currently available for inclusion in computer entertainment systems are MIDI-based systems which provide only a small subset of the features included in many of the stand-alone software "sequencers" used in the music industry. A sequencer is basically a software tape player, except that instead of converting stored sound waveforms into sound, a sequencer interprets stored musical performance data and issues corresponding commands to sound generating hardware. An existing music industry product which represents the most advanced sequencer technology currently available is Performer, a product of Mark of the Unicorn, Cambridge, Massachusetts.
The MIDI-based music systems in existing computer entertainment systems, although software controllable unlike the above-mentioned stand-alone sequencer, are relatively limited. The limitations with these systems derive primarily from the structure of existing music files, coupled with the relatively simple architecture of the way the music playback system operates. Existing music sound files, while containing multiple channels of performance data, do not contain any provisions for branching or conditional messages, nor can the playback system be re-configured to respond differently to the performance data. Thus, existing systems cannot make compositional decisions based upon action of the game, but can only play the music in one way as determined by the stored performance data. By contrast, a motion picture sound track is scored by a composer in response to what the composer aesthetically perceives about the film's events, action, and movement.
In scoring a motion picture, a composer uses a combination of live musicians and electronic instruments to assemble the eventual sound track. The composer creates and modifies the music while watching the motion picture, thus creating a sound track which is an integral part of the viewing experience. Often, several versions of the music are composed, and the director of the motion picture selects the final version during post-production by using segments of music from each of the earlier composed versions.
In existing computer entertainment systems, the music is precomposed and stored as musical sequences (human composed music tends to be far more evocative than algorithmic music). These sequences are generally based on a standard MIDIfile format or variations thereof. A MIDIfile contains performance data representing a sequence of musical notes to be played. The performance data specifies the time between one note and the next, and the MIDI channel (i.e. instrument selection, such as trumpet or violin) for each note. There may also be other performance parameters specified by the data, such as volume changes which are to occur at a particular place in the music. A standard MIDI sequence essentially describes a musical performance, performed by up to sixteen different instruments playing in parallel, which is played in essentially a linear manner from beginning to end.
MIDI performance data is interpreted by what is basically a software tape player, or sequencer. A sequencer reads through a MIDIfile, pausing between each MIDI message for a period of time specified by the data in the file. Each time it proceeds to a new message, the message is decoded and an appropriate command is sent to a synthesizer to produce the actual sound.
In the existing systems, the sequencers read through MIDI sequences in the simplest way: they start at the beginning of the sequence and read forward until they reach the end. (In some cases, when a sequencer reaches the end of a sequence, it automatically restarts from the beginning. This is called "looping", and allows a sequence to be repeated indefinitely.) The sequencer path is essentially linear, in that no provisions are made within the MIDIfile format for branches, and the sequencer is not capable of jumping to arbitrary time points within the sequence. Thus, in existing systems, the ability of the host to control the sequencer, and thereby compose the actual music played is limited to starting a sequence at the beginning, and stopping it at any time before it reaches the end.
Considering these limitations, it is easy to see why the musical flow suffers in existing computer entertainment systems when the music is required to change from one sequence to another. For example, suppose that there is a high-energy fight scene occurring in the game which, at any time, may end in either victory or defeat. In existing systems there would likely be three music sequences: fight music (looped), victory music, and defeat music. When, the fight ends, the fight music would be stopped, and either victory or defeat music would be started. The switch from the fight music to the victory or defeat music occurs without taking into account what is happening in the fight music leading up to the moment of transition. Any musical momentum and flow which had been established is lost, and the switch sounds abrupt and unnatural.
A further limitation of existing systems is that the synthesizer responds to the MIDI commands in the most direct fashion; it plays the notes which the composer has sequentially placed in the MIDI sequence. This literal interpretation of MIDI commands precludes an entire category of dynamic music composition, i.e. allowing the computer entertainment system to make dynamic changes to the way in which the performance data are interpreted, resulting in aesthetically appropriate variation in the music while the game is playing, thereby tailoring the texture, mood, and intensity of the music to the action in the game.
Thus, existing music systems do not provide the ability for the computer entertainment system to tell the music system how to intelligently and artistically respond to the events and action of the game. There is needed a music and sound effects system which can be included in a computer-controlled sound system including a computer entertainment system, and which creates natural and appropriate music composition that changes dynamically with the events and action of the game in response to commands from the sound system.